-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-34887][PYTHON] Port Koalas dependencies into PySpark #32386
Conversation
dev/requirements.txt
Outdated
# Optional dependencies in pandas-on-spark. | ||
mlflow>=1.0 | ||
plotly>=4.8 | ||
matplotlib>=3.0.0,<3.3.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this file used for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's used for setting up a development environment for PySpark.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure we should add them here because even dependencies for SQL or ML modules are not here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this file is not used anywhere .. it's just for dev setup :-).
python/setup.py
Outdated
'pandas>=0.23.2', | ||
'pyarrow>=0.10', | ||
'numpy>=1.14,<1.20.0', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we should install them by default but should be extras.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They are moved to extras_require
.
Does it mean when users install pyspark, pandas_on_spark doesn't work by default
?
python/setup.py
Outdated
'pandas.mlflow': ['mlflow>=1.0'], | ||
'pandas.plotly': ['plotly>=4.8'], | ||
'pandas.matplotlib': ['matplotlib>=3.0.0,<3.3.0'], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure we should list these here? cc @HyukjinKwon
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we could:
extras_require={
...
'pandas_on_spark': [
'pandas>=0.23.2',
'pyarrow>=0.10',
'numpy>=1.14,<1.20.0',
]
...
Yeah, I think we should remove them for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might need to add some description in [1] to add the dependencies info for pandas_on_spark.
[1] https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Optional dependencies for pandas-on-spark have been removed.
Dependencies for pandas-on-spark have been moved to extras_require
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Descriptions are added to install.rst
.
Shall we call it Pandas-on-spark
in the document?
dev/requirements.txt
Outdated
@@ -6,3 +6,13 @@ pydata_sphinx_theme | |||
ipython | |||
nbsphinx | |||
numpydoc | |||
|
|||
# dependencies in pandas-on-spark. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@xinrong-databricks
Do you intend to include dev dependencies in this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, this file is only used when setting up a dev env for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this PR includes dev dependencies.
Kubernetes integration test starting |
Kubernetes integration test status failure |
Test build #138053 has finished for PR 32386 at commit
|
'pandas>=0.23.2', | ||
'pyarrow>=0.10', | ||
'numpy>=1.14,<1.20.0', | ||
], | ||
extras_require={ | ||
'ml': ['numpy>=1.7'], | ||
'mllib': ['numpy>=1.7'], | ||
'sql': [ | ||
'pandas>=%s' % _minimum_pandas_version, | ||
'pyarrow>=%s' % _minimum_pyarrow_version, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: pyarrow minimun version is introduced in _minimum_pyarrow_version, but if we introduce the pyarrow>=0.10
for pandas_on_spark in here, maybe we could change the _minimum_pyarrow_version
to something like _minimum_sql_pyarrow_version
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! There is only one usage of _minimum_pyarrow_version
. How about removing the variable and using the value instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we can use the same lower bound, I prefer just use _minimum_pyarrow_version
directly as suggestion in #32386 (comment)
'pandas_on_spark': [
'pandas>=%s' % _minimum_pandas_version,
'pyarrow>=%s', % _minimum_pyarrow_version,
]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good.
python/setup.py
Outdated
'pandas.mlflow': ['mlflow>=1.0'], | ||
'pandas.plotly': ['plotly>=4.8'], | ||
'pandas.matplotlib': ['matplotlib>=3.0.0,<3.3.0'], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might need to add some description in [1] to add the dependencies info for pandas_on_spark.
[1] https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies
Kubernetes integration test starting |
Kubernetes integration test status failure |
Test build #138088 has finished for PR 32386 at commit
|
python/setup.py
Outdated
], | ||
'pandas_on_spark': [ | ||
'pandas>=%s' % _minimum_pandas_version, | ||
'pyarrow>=0.10', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The pandas on Spark is based on PySpark SQL with pandas UDF which won't work with pyarrow < 1.0.0
. We should use the same lower bound.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. The same lower bound is used.
dev/requirements.txt
Outdated
# Optional dependencies in pandas-on-spark. | ||
mlflow>=1.0 | ||
plotly>=4.8 | ||
matplotlib>=3.0.0,<3.3.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure we should add them here because even dependencies for SQL or ML modules are not here.
`pandas` 0.23.2 Required for Pandas-on-spark | ||
`pyarrow` 0.10 Required for Pandas-on-spark | ||
`Numpy` 1.14(<1.20.0) Required for Pandas-on-spark |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pandas-on-Spark
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or "PySpark APIs on Spark" which will be the official name. Either one should be fine because ML
is used over MLlib DataFraem-based API
as its official name (see http://spark.apache.org/docs/latest/ml-guide.html#announcement-dataframe-based-api-is-primary-api)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
May I confirm the official name is PySpark APIs on Spark
? or pandas APIs on Spark
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pandas-on-Spark
is used for now to keep consistent with ML
for example.
Test build #138113 has finished for PR 32386 at commit
|
Kubernetes integration test starting |
Kubernetes integration test status failure |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Otherwise, LGTM.
Merged to master. |
What changes were proposed in this pull request?
Port Koalas dependencies appropriately to PySpark dependencies.
Why are the changes needed?
pandas-on-Spark has its own required dependency and optional dependencies.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manual test.
Keyword:SPARK-34849